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Abstract 

Computers and the internet have been utilized as viable avenues for public health education delivery. Yet the 
effectiveness, e.g., behavior change, from use of these tools has been limited. Previous reviews have focused on 
single health topics such as smoking cessation and weight loss. This review broadens the scope to consider 
computer-assisted learning across the field of health. Of the 99 publication, 58 were selected for analysis. First, the 
literature was qualitatively summarized. The studies could not be compared directly due to the specifics of each 
study. However, a commonality emerged to allow the meta-analysis of the quantitatively pooled results of 
randomized control trials (RCTs). The meta-analysis indicated that, in the literature analyzed, the use of computers 
provided a positive effect (general improvement) through changes in knowledge, attitudes, and/or behavior. This 
critical review will be constructive for the development of more comparable theory-driven, evidence-based 
educational programs via the internet in the future. 
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Introduction 

Despite the use of computers and the internet since 
the 1980s to deliver health messages to the public, 
there is limited empirical evidence that computer- 
assisted learning via the internet affects change in 
health behaviors.*'^ This may not concern commercial 
website developers who reach out to individual 
internet users in order to generate mass sales. 
Whereas, the aim of public health practitioners is to 
reach out to the mass of internet users to help 
individuals. In terms of this altruistic goal, the 
internet is an ideal educational tool because it is 
“free” and on demand for everyone. It can be 
continuously updated as scientific evidence evolves, 
adapted to different cultures, translated into all 
languages, and used to reduce the costs of face to 
face teaching and media production even as the 
number of users increases.^’ ® 

In theory, the internet enables individual users to 
overcome their psycho-social barriers to behavioral 
change by providing anonymity to users who are 
uncomfortable with their health issues while at the 
same time providing public health practitioners with 
the potential to access and monitor their target 
audiences. Specific to public health, the internet 
allows the scaling of a health message for 
populations down to individuals without losing its 
original effectiveness.^ Previous reviews and 
countless effectiveness and feasibility studies have all 
come to the same conclusion regarding the perceived 
usefulness of computer-assisted learning via the 
internet. It is a viable strategy for health education.* '® 
However, the evidence tends to stop here. 

The current trend in computer-assisted learning via 
the internet is to include interactive components that 
engage individuals and tailor information to their 
needs, • but these programs have been 

slow to emerge in public health primarily because 
many practitioners, e.g., community health nurses, 
nutritionists, and educators, lack the funding, 
computer skills, time and/or advanced programming 
tools to develop contemporary internet products. 
Regardless if they have the creative talent, public 
health practitioners do not usually have the time to 
design nor the staff to manage effective websites. 
Compounding the challenges for newcomers to the 
computer programming arena is the existing literature 
on computer-assisted learning which has overused 
the term “interactive” to the point of deception. To 
label a product interactive does not prove it is 
effective. Therefore, the literature, which should be 
guiding public health practitioners to correctly 


estimate the costs and skilled man-hours needed to 
develop and maintain computer-assisted learning 
programs via the internet, does not. 

Unfortunately, public health practitioners are left 
using unsophisticated tools to develop first 
generation programs simply to satisfy the 
requirement to provide resources to the public. The 
result is a persistent contribution to the internet as an 
information dump which inadvertently requires users 
to be highly self-motivated to wade through pages 
and pages of web text in order to find information 
that is personally relevant. The advent of powerful 
search engines has helped to direct users’ attention, 
but good and valid health information websites are 
drowned out by commercial sites and 
advertisements."' Even relatively modern websites 
fall into the category of first generation programs if 
they contain no narration, no navigation instructions 
and no multi-media.'^ These programs have not been 
found to be significantly better than traditional media 
materials;^*'** are written at a tenth grade reading 
level or higher;*^ and worse yet, have the added 
stigma of doing “more harm than good.”"'’ ** 

Second generation programs, on the other hand, may 
be more likely to be read and remembered*"' because 
they provide a more friendly user interface with 
graphics, audio, video, animation, guiding links and 
communication tools making them more competitive 
with commercial sites. However, these technological 
bells and whistles do little to increase readers’ 
viewing beyond seven to thirteen minutes per 
website*’ **’ ** Alternatively, registering readers and 
requiring them to log onto secure sites is the future, 
today.*®’ ** These websites employ survey tools to 
measure, for example, knowledge, attitudes and 
behavior, but they are still far from being interactive 
if no health message is tailored to individual users’ 
needs.*® Researchers are finding that if participants 
do not receive something substantive in return for 
their time, they will quickly drift away.*’ **’ **’ ** "** 

From the studies presented in this review, there are 
lessons to be learned, like the aforementioned, which 
can guide, in toto, modern practices in health 
education. To help program developers move beyond 
process analyses, we present a meta-analysis of the 
change in knowledge, psycho-social variables 
(attitudes), and behaviors from available randomized 
control trials (RCTs) of second generation programs 
developed across the field of public health. In so 
doing, we hope to encourage a more critical dialogue 
regarding computer-assisted learning via the internet. 
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Methods 

To generate an initial list of studies, we searched 
independently PubMed, Google and relative journals 
for adult health education interventions using 
technology. Studies with youth, defined as 
participants under the age of 18 years, were excluded 
as well as studies with families which included youth. 
The key words used were: computer; interactive; 
internet; multimedia; and web. These key words were 
then combined with health topics revealed through 
the search. As programming trends were identified in 
the literature, publications prior to 1999 were 
excluded from the search to eliminate as many 
knowledge-based, first generation programs as 
possible and to eliminate, for example, reports on 
flow-sheets; automated telephone calls; and database 
management systems, which have been sufficiently 
reviewed in the past.^’ This initial search yielded 
99 English language publications (53 full-text articles 
and 46 abstracts) ranging in topics from asthma to 
verrucae. See Diagram 1 . 

Of the 99 publications found: 13 were review 
articles; 28 were narratives or qualitative studies of 
effectiveness, feasibility and process; and 58 were 
quasi-experimental RCTs. We performed a 
preliminary qualitative analysis of these 58 RCTs to 
identify common themes in research objectives, 
design, outcome measures, results and stated 
conclusions, which is an approach slightly different 
from other reviews of the literature"*’ ' ' and 

websites. Instead of generating a summary of 
publications on computer-assisted learning in one 
health discipline, we report who is doing what, when, 
where and how across public health to aid 
practitioners and researchers endeavoring to develop 
educational programming via the internet. 

As expected, we found that every study was 
inherently different because of the specific nature of 
each health topic, which prevented any direct 
comparisons. However, we also found that all the 
RCTs sought, in general, to measure change in 
knowledge and/or psychosocial variables (attitudes) 
and/or behavior in a randomly assigned intervention 
group exposed to a new or improved computer- 
assisted learning program. The conclusions reported 
by the investigators of these studies could be 
categorized into either significant improvement or no 
improvement of the three measured outcomes and 
allowed for binomial testing of the null hypothesis 
that exposure to computer-assisted learning does not 
result in any improvement. We hypothesized that the 


combined data would show evidence of the 
effectiveness of computer-assisted learning to 
improve, in general, one or more of the following: 
knowledge of any health topic, attitudes towards any 
healthy behavior, or any targeted health behavior. 

For the meta-analysis, the RCTs designed to test the 
use of cell phones or personal digital assistants,"**’ "*® 
although something to be considered in the future,^^ 
were excluded as not fitting the traditional definition 
of computers. Computer-assisted learning programs 
provided at kiosks**’ *®’ *^ and computers situated in 
physician offices, social services offices, or public 
spaces ’ ’ were deemed unnatural settings for 
computer use and excluded. RCTs which only 
measured perceived effectiveness of a program and 
not change in knowledge, psycho-social variables 
(attitudes) or behavior were also excluded. Data from 
29 RCTs remained and was separated into three 
binomial tables to record significant improvement 
(yes or no) in knowledge, attitudes and behavior to be 
used in a preliminary meta-analysis. 

The frequency of observations of improvement and 
the probability of observing no improvement were 
determined using the Fisher Exact Test. Then each 
RCT resulting in significant improvement in any one 
of the three measured outcomes was identified 
through the transformation of data into one category 
of “General Improvement” (yes or no). This more 
robust category enabled us to test the dependency of 
achieving any Improvement (n=32) upon the use of 
training (yes or no) and/or incentives (yes or no) in 
the research design (n=6). Fischer Exact testing for 
categorical variables was again chosen for the 
statistical analyses of the data over Pearson’ s Chi- 
square test because it has more power to detect 
dependency between small samples.^** 

Next, because culturally diverse samples may not be 
appropriate for comparison,*® the international studies 
were separated from the domestic studies. Again, 
Fischer Exact test was used to determine if General 
Improvement was dependent upon the sampling 
being from outside the United States (yes or no). In 
this case, Pearson’ s Chi-square test was also used to 
verify the results. Last, because the international 
studies were observed to have shorter follow-up 
times (0-90 days) and because the literature noted 
that reporting immediate improvements may not be 
enough evidence of effectiveness,^ we used the 
Cochran-Mantel-Haenszel test to determine if 
Improvement was dependent upon the length of time 
of the studies. 

Preliminary hypothesis testing of the initial data set 
(1999 - May 2008) was encouraging and led us to 
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perform a secondary search for articles published 
through December 2008. From this secondary search, 
the pooled data from 18 more RCTs of computer- 
assisted learning across public health were added to 
the meta-analysis. Data were managed using Excel 
2000 (Microsoft Corp., Redmond, WA) and were 
analyzed using R 2008 (version 2.0, www.r- 
proiect.org) . See Tables 1 and 2 for the results. 

Results 

The literature review of computer-assisted learning 
across the field of public health revealed a top 10 list 
of topics including: alcohol consumption awareness; 
cancer (breast, colorectal, and prostate), 
cardiovascular disease; diabetes (care and education); 
eating disorders; general health including lifestyle 
behaviors; general nutrition (dietary behavior, weight 
loss, and nutrition education); HIV/AIDS; physical 
activity; and smoking cessation. Four types of 
research designs were identified such that 
intervention groups were exposed to a specific 
computer program and compared to control groups 
that either received no exposure to a program at all, 
exposure to a placebo program typically providing 
generic information, or exposure to alternative media 
or standard care. The fourth type of design tested the 
effect of an improved program (intervention) 
compared to an existing program (control). For the 
meta- analysis, the results of exposure by an 
intervention group to a new or improved computer- 
assisted learning program were pooled as data. 

Because of the number and quality of international 
RCTs found in the literature, they could not be 
ignored. In the final analysis, the majority of the 
international studies published in English came from 
the Netherlands; Australia and New Zealand; 

Canada; and the United Kingdom. Of the peer- 
reviewed journals cited in this review, the following 
offered the greatest variety of studies including 
RCTs: Addiction; Annals of Behavioral Medicine; 
Diabetes, Technology and Therapeutics; the 
International Journal of Eating Disorders; the Journal 
of Medical Internet Research; the Journal of Nutrition 
Education and Behavior; Patient Education and 
Counseling; and Obesity Research. Furthermore, the 
dominant authors in the field provided a considerable 
history of program development over the last decade 
as well as a peek into the future of computer-assisted 
learning via the internet. Those who were recognized 
included: R. J. Bensley; J. Brug J; M. K. Campbell; 

R. E. Glasgow; E. H. Jackvony; W. Kroeze W; K. 
Kypri K; E. Lehmann E; S. Linke; A. Oenema A; V. 


J. Strecher; D. F.Tate; M. W.Verheijden; R. A. 

Winett RA; and R. R. Wing. 

These and other researchers reported a number of 
“firsts” in computer-assisted learning from 1999 
through 2008. In 2001, Oenema, Brug, and Lechner 
reported being the first to empirically evaluate a 
second generation computer tailored intervention^' 
which measured changes in determinants of behavior, 
i.e. awareness and intention. The intervention group 
received tailored nutrition education via the internet 
and the control group receiving a non-tailored 
information letter. A significant increase in 
awareness and intention to make dietary changes was 
found with the intervention group. Also in 2001, 

Tate, Wing and Winnett reported being the first to 
evaluate an internet behavior therapy program.""* The 
intervention group received an educational website 
plus support through email, diaries and bulletin 
boards, and the control received only an educational 
website. The investigators measured change in 
weight and physical activity. They found a greater 
number of participants in the intervention group lost 
5% of their weight and maintained their weight loss 
for up to 6 months, but found no significant 
difference in physical activity between the two 
groups. 

In 2004, Womble and colleagues reported being the 
first to evaluate a commercially available weight loss 
program on the internet to alternative media, i.e. a 
weight loss manual.^^ The investigators measured 
change in cardiovascular disease risk factors and 
weight. After 52 weeks, they found no significant 
improvement in risk factors and reported greater 
weight loss by the control group using the manual. 

In 2005, Miller and colleagues reported being the 
first to study the effect of computer-assisted learning 
compared to standard care focusing on health 
screening.*® Change in knowledge and behavior was 
measured, but the results were inconclusive. In 2006, 
Suminski and Petosa reported being the first to study 
the use of Social Cognitive Theory in computer- 
assisted learning compared to no exposure to a 
program.^^ After nine weeks, knowledge and self- 
regulation increased in the intervention group, but 
there was no change in the constructs of self-efficacy 
and social support. 

In 2007, Polzien, Jakicic, Tate and Otto reported 
being the first to study the addition of computer- 
assisted learning with intermittent and continuous 
doses to standard care.^® Differences in levels of 
exposure were not found to be statistically 
significant; however the investigators argued that the 
results may have been clinically significant, and 


International Electronic Journal of Health Education, 2009; 12:72-85 


4 


A Critical Review of Computer-assisted Learning in Public Health, 1999-2008 


Corda & Polacek 


deserved further investigation. Most recently, in 
2008, Riper, Kramer, Smit and colleagues reported 
being the first to test a multi-component, interactive 
self-help program to alternative media, i.e. a 
brochure.^"^ Behavior change, measured as decreased 
mean weekly alcohol consumption to normal was 
found to be significant for the intervention group at 
six months follow-up. 

The conflicting results, as seen above, have been 
reported throughout the literature and reinforced the 
conclusion of a 2004 review of the literature by 
Verheijden and colleagues that “as of yet computer- 
based interventions have not been the magic 
breakthrough they were hoped to be.”^’ * However, 
investigators of only four of the RCTs reported using 
power equations of which none reached 90% power 
(a = .05)*°’ **’ ^*’ "** indicating a problem in how and 
what was being measured. When sample sizes were 
reported too small and when attrition was reported 
too great, the conclusions were that the true effect 
was too difficult to measure.^’ *°’ Therefore, results 
in the literature across the field of public health may 
be underestimated. 

Also noted was that less than half of investigators in 
this review reported using theories or models to guide 
their research designs. However, the majority of the 
studies using theories and models have come about 
within the last few years, after 2006, which indicates 
a positive trend. The theoretical constructs reportedly 
used most often were from; Cognitive Behavior 
Theory;^"* Goal Setting;*’"** Motivational Theory;"** the 
Transtheoretical Model;^’ *°’ Health Belief 
Model;*®’ ^’Theory of Self-control;^"* and most 
recently, the Social Cognitive Theory.*®’^® The media 
uses and gratification paradigm was used for 
perception of use of media material,^® and when no 
one theory of behavioral change would do, multiple 
constructs from multiple theories were utilized as 
needed.^*’ 

In summary, because the qualitative analysis of the 
literature proved to be inconclusive, a meta-analysis 
to increase effect size was performed. From the 
observed values of the 47 RCTs in the final meta- 
analysis; knowledge was a measured outcome in ten 
of the studies and reported to significantly increase in 
nine of them (90%); attitudes was a measured 
outcome in 20 of the studies and reported to 
significantly improved in 15 of them (75%); and 
behavior was a measured outcome in 35 of the 
studies and reported to significantly increase in 20 of 
them (57%). Assuming the studies were independent 
of one another, the probabilities of not observing any 


improvement in knowledge, attitudes and behavior 
were all significant. See Table 1. 

To determine if the use of training or incentives had 
an effect on the outcomes, a new variable, “General 
Improvement,” was created. The result of the Fisher 
Exact test comparing the category of General 
Improvement to Training was p=0.6484 and for 
General Improvement to Incentives was p=1.0. These 
results indicated that the outcomes of the observed 
studies were not dependent upon the use of training 
or incentives. 

Next, whether or not international sampling had an 
effect on General Improvement was investigated. The 
result of the Fisher Exact test was p=0.04854 (a = 
0.05) which indicated dependency. The result of a 
Pearson’s Chi-square test for dependency was 
p=0.05675 (a = 0.05), which approached 
significance. Together, the evidence for dependency 
is not strong. 

When the differences between the international 
studies (n=17) and the domestic studies (n=30) were 
examined more closely, we found that besides 
cultural and socio-economic differences more of the 
domestic studies (57%, n=17) had a longer follow-up 
time than the international studies (35%, n=6). Eor 
convenience, we chose 90-days follow-up as a cut off 
to categorize the RCTs since 50% of the studies 
(n=23) fell into the category of less than 90-days in 
length and 50% of the studies fell into the category 
(n=24) of 90-days or more in length. The results of a 
Cochran-Mantel-Haenszel test for independence of 
Improvement on length of time, set conditionally at 
90 days, was p=0.1012 (a = 0.05). This finding was 
not significant even though an analysis of the initial 
data (not reported) indicated some unknown 
differences between the international and domestic 
studies which could not be identified. The analysis 
was stopped here because pooled data was being 
utilized for this analysis. Thus the findings were not 
reliable. 

Discussion 

This meta-analysis of computer-assisted learning 
across the field of public health is the first of its size 
and design. Even though our data set is limited, using 
pooled output data is a simple yet valid method of 
analysis from which we found strong quantitative 
evidence. The probability of observing no effect on 
the measured outcomes was very small (p < 0.0001); 
therefore, we conclude that computer-assisted 
learning can improve knowledge, attitudes and 
behavior. 
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However, a comparison of dissimilar research 
designs limits our discussion,"^ and without more raw 
data, we must turn to our qualitative analysis to 
support the need for further development and testing. 
Unfortunately, we could find no starting point in the 
literature for researchers endeavoring to develop 
evidence-based computer-assisted learning programs. 
In education, this would be at least a list of best 
practices, but the lack of these practices may account 
for why the use of the internet currently in public 
health education is still being described as a “viable 
adjunct,”*’ *^^^ a “novel opportunity,”^"* or having 
“potential benefit.”^’ As stated before, this random 
evidence of effectiveness falls short of what 
researchers need to move forward. However, the 
literature supports the emerging use of theories and 
models to provide a way to consistently measure 
outcomes between studies in the future. 

For example, the Transtheoretical Model was found 
to be most prevalent in the literature and would seem 
to be most beneficial®** because computer programs 
“can be used as a counseling tool when strategies are 
outlined for specific stages of change”®^ and designed 
as computer algorithms. *®’ "*®’ ®®’ ®‘* The 

Transtheoretical Model also lends itself well to the 
development of expert tailored systems that provide 
individualized feedback."*^ Since 2001, tailoring has 
been recognized as needed in computer-assisted 
learning^*’ "***’ "**’ ®*’ ®® to identify and provide greater 
assistance to those coming from the most severe 
baseline,^*’ ®‘*’ ®®’ ®® i.e. lowest stage, highest risks, 
relevant family histories, or seriousness of a case.®’ *^’ 
31,43,60,67-69 djnical Setting, use of stage of change 
theory is aiding practitioners with triaging clients.*^ 

However, effective tailored programming requires 
extensive interactivity,^**’ which is dependent upon 
available technologies not only to public health 
practitioner but also to their target audiences. 
Interestingly in this meta-analysis, a 
disproportionately low number (< 10%) of the RCTs 
sampled domestic, underserved populations. • • ■ • 
While anecdotal evidence suggested that those who 
have low socio-economic status have greater 
satisfaction with computer-assisted learning than high 
socio-economic status, possibly due to their greater 
need,®’ ®® portions of this population may still not 
have access to computers, use the internet, or 
computer literacy skills. This finding brings into 
question the assumptions that program developers in 
public health may have regarding the available 
resources of their target audiences, and requires 
further investigation.®®’ ^* 


What we did find currently being explored in the 
literature included user input for preference of 
bulletin boards, chat groups, blogs and on-line diaries 
to increase participant engagement. ’ ’ In 
community settings, programs which adapt to 
multiple-literacy levels ’ ’ were being considered, 
and in clinical settings, telephone or email contact 
with human counselors has been found to be most 
beneficial This may seem surprising, but the 

belief that the development of computer-assisted 
learning is to replace people is now “outdated.”®^ 
Programs today are being designed to compliment 
standard care,*^ i.e. to reduce the burden on staff,*®’ **’ 

’ ’ and to save time and money. For example, 
providing personal assistance to everyone with 
technology, instruction and orientation is becoming 
more common because it is thought to retain users**’ 
30,41,57,73,76 Ujjfortuuately, in this meta-analysis we 
were unable to stratify the data by demographics that 
relate to computer literacy, such as age and socio- 
economic status, but our finding that General 
Improvement was independent reflected the findings 
of Tate, Jackvony and Wing who reported that 
computer assistance did not bias the results of their 
study."** These findings, while limited, underscore the 
need to determine the correct “dose” of computer- 
assisted learning to social interaction. *’’ ^"*’ ®®’ "**’ ®*’ ’’’ 

Conclusion 

There appears to be ample room for dialogue among 
public health practitioners who are developing 
computer-assisted educational programs via the 
internet. We found that the literature from within 
individual health disciplines is not enough to 
substantially improve current practices, and, 
therefore, broaden our scope to look critically at the 
available research across the field of public health. 
Additionally, we found evidence that, in general, 
computer-assisted learning programs do improve 
knowledge; attitudes; and behavior, but that there is a 
need for more theory -driven, evidence-based research 
that can be replicated. It is hoped the contribution of 
this critical review of the literature regarding the use 
of computer-assisted learning in public health can 
inform those who endeavor to develop educational 
programming via the internet so that they may leap 
ahead of current practices. 
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Figure 1. Inclusion/Exclusion Process for RCTs Included in the Final Meta-analysis 
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Table 1. Hypothesis Testing of Knowledge, Attitudes, and Behaviors 


Pooled Output Data 

Frequency of 
Observations 
(n = 47) 

Per Cent 
Improvement 
Observed 

Fisher Exact Test 
Probability (p), 

95% Confidence Interval (Cl) 

Knowledge 

No Improvement 
Significant Improvement 

1 

9 

90% 

p = 9.1e-9“ 

Cl = 55.5%, 0.99.8% 

Attitudes 

No Improvement 
Significant Improvement 

5 

15 

75% 

p = 9.5e-12“ 

Cl = 50.9%, 91.3% 

Behavior 

No Improvement 
Significant Improvement 

15 

20 

57% 

p = 7.3e-12“ 

Cl = 39.4%, 73.7% 


Note: 

“The results are signifieant (a = 0.05); therefore, we reject the null hypothesis that exposure to computer-assisted 
learning does not result in improvement. 


International Electronic Journal of Health Education, 2009; 12:72-85 


13 




A Critical Review of Computer-assisted Learning in Public Health, 1999-2008 


Corda & Polacek 


Table 2. Test for Dependency Testing of General Improvement on the use of Training, the use of Incentives and 
the Sample Population 


Combined Data 

General 

Improvement 

Dependency 

Testing 

(Probability = p) 

No Training (n = 42) 

- 


No Improvement 

13 


Significant Improvement 

29 

p = 0.6484 

Training (n = 5) 

- 

(a = 0.05) 

No Improvement 

2 


Significant Improvement 

3 


No Incentive (n = 42) 

- 


No Improvement 

14 


Significant Improvement 

28 

p= 1.0000 

Incentive (n = 5) 

- 

(a = 0.05) 

No Improvement 

1 


Significant Improvement 

4 


Domestic (U.S.) Studies 

- 


No Improvement 

13 

p = 0.04854“ 

Significant Improvement 

17 

(a = 0.05) 

International Studies 

- 


No Improvement 

2 

= 0.05675’’ 

Significant Improvement 

15 

(a = 0.05) 


Notes: 

“The results of the Fisher Exact test are significant. 

’’The results of the Pearson’s Chi-square test approach significance. 
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